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process.  » 
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"RELIABILITY  GROWTH"  AS  AN  ARTIFACT  OF  RENEWAL  TESTING 


by 

William  S.  Jewell 


1 ■ INTRODUCTION 

"Reliability  growth"  refers  to  the  empirically-observed  fact  that 
the  performance  of  a complex  system  shows  improvement  as  the  newly-designed 
system  moves  into  the  development  program  and  is  tested  under  actual  or 
simulated  operational  conditions.  This  improvement  results  primarily 
from  the  identification  and  correction  of  initial  design  and  engineering 
deficiencies  [6],  but  may  also  be  due  to  improved  familiarity  in  the  field 
by  operating  personnel,  revised  operating  and  maintenance  procedures,  etc. 
Analysis  and  extrapolation  of  reliability  growth  is  of  critical  importance 
to  both  the  manufacturer  and  the  purchasing  agency  because  of  the  need  to 
balance  (1)  the  high  costs  of  making  engineering  changes;  (2)  the  costs  and 
delays  in  extended  testing;  (3)  the  operational  support  costs  of  maintenance 
and  replacement;  and  (4)  the  need  for  operational  readiness  and  reliability. 
Currently,  the  U.S.  Air  Force  is  experimenting  with  a new  procurement 
procedure,  the  Reliability  Improvement  Warrant  [3],  [8],  [12],  under  which 
financial  incentives  are  offered  to  contractors  "to  design  and  produce 
equipment  which  will  have  a low  failure  rate  as  well  as  low  repair  costs 
after  failure  due  to  field/operational  use"  [12], 

Beginning  with  [14],  there  have  been  many  models  proposed  for 
reliability  growth  [5],  [7].  The  two  major  approaches  [11]  are:  (1)  to 
assume  that  observable  improvements  occur  only  in  discrete  phases,  as 
major  redesign  efforts  follow  testing;  (2)  to  assume  that  a continuous 
"learning  curve"  of  reliability  improvement  applies  during  the  development 


program.  The  most  successful  model  of  the  second  type  is  due  to 
J.  T.  Duane  (see  [7]);  it  assumes  that  the  failure  process  is  time- 
varying  Poisson,  with  cumulative  mean  failure  rate  a decreasing 
algebraic  (Weibull)  function  of  system  total  operating  time  (including 
fixes  and  replacements). 

The  purpose  of  this  paper  is  to  suggest  that  some  of  the  "reliability 
growth"  supposedly  observed  during  testing  programs  may  be  an  artifact  of 
the  parameter  chosen  for  measurement,  and  the  statistic  used  to  estimate 
that  parameter,  particularly  in  short-duration  field  performance  demonstra- 
tion programs  after  major  design  developments  have  been  implemented. 

Our  approach,  suggested  by  I.  Shimi,  is  to  assume  that  the  underlying 
failure  process  is,  in  fact,  stable , and  to  show  that  reasonable  estimators 
are  biased  functions  of  the  testing  interval  length.  These  results 
suggest  that  it  may  be  difficult  to  separate  true  reliability  growth  from 


the  measurement  process. 
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2.  TESTING  MODEL  AND  PERFORMANCE  ESTIMATORS 

We  assume  that  the  failure  mechanism  of  the  system  generates  a 
renewal  process,  in  which  successive  random  intervals  between  failures, 
X^,X2»  ...  are  independent  and  identically  distributed  samples  from  a 
distribution  function  F , with  density  f(x)  = dF(x)/dx  , complement 
F = 1 - F , and  moments  nu  = E{X^}  . Consider  a single  system 
placed  on  renewal  testing  for  a total  operating  time  of  t hours;  during 
this  interval  a random  number  N(t)  of  failures  and  instantaneous  fixes 
occur,  with  the  last  failure  occurring  at  epoch  = X1 + X2 + ' * ’ + XN( 

YN(t)  - t > (Figure  D • 


h — xi~ — x2 — H H — ^ — H 


-*b 


Start  of  test 


time 


End  of  test 


Figure  1 


Renewal  Testing  Scheme 
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The  first  performance  estimator  to  be  considered  might  be  the 
empirical  failure  rate 

(1)  *(t>  = (t  > 0)  . 

Presumably,  if  there  is  "reliability  growth,"  N(t)  will  be  "concave 
down"  [2],  or  <|i(t)  will  be  decreasing  with  t . [13]. 

A more  useful  performance  measure,  and  the  one  upon  which  the 
Reliability  Improvement  Warranty  is  based,  is  the  mean-time-between-failure 
(MTBF) . Depending  upon  one's  statistical  sophistication,  there  are  three 
estimators  that  suggest  themselves  from  Figure  (1) : 

1 . The  Naive  MTBF  Estimator 

Since  there  are  N(t)  complete  samples  observed,  the 
sample  mean  lifetime  should  be  used  as  MTBF  estimator: 

N(t) 

(2)  1^(0  = l Xi/N(t)  = YN(T)/N(t)  , (t  > 0) 

assuming  that  at  least  one  failure  occurs  in  (0,T]  (N(t)  > 0)  . 

2 . The  Clever  MTBF  Estimator 

The  alert  reliability  engineer  will  realize  that  the 

residual  interval  t - ’ rePresents  incomplete  sample 

information  that  should  not  be  discarded.  Further,  since 

exponentially-distributed  lifetimes  are  of  important  practical 

interest,  and  it  is  known  that  these  lifetimes  are  "memoryless," 

one  might  argue  that  the  incomplete  sample  is  similar  to  an 
s t 

(N(t)  + 1)  sample,  so  the  MTBF  estimator  should  be: 
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(3)  u2^  **  N(t)  + 1 > 

which  is  defined  even  if  there  are  no  failures. 

3.  The  Sophisticated  MTBF  Estimator 

A sophisticated  analyst  will  have  read  the  literature  on 
renewal  testing  (see  [5]  and  [9]),  and  will  realize  that  the 
maximum  likelihood  estimator  of  MTBF  in  the  exponential  case 
is  the  total-time-on-test  divided  by  the  number  of  complete 
samples,  or 

(4)  y3(t)  = t/N(t)  , (t  > 0) 
again  defined  only  if  N(t)  > 0 . 

In  examining  the  above  estimates,  it  is  clear  that  U^Ct)  = l/4i(t)  » 
P3(t)  > u^(t)  , and  u3(t)  > > but  that  the  actual  sample  values 

may  follow  rather  complicated  paths  as  functions  of  the  testing  interval 
t . 

In  what  follows  we  shall  take  two  approaches: 

(1)  Find  the  expected  value  of  each  estimator.  This  reflects  the 
mean  value  of  the  sample  function  for  a single-system  test,  and 
also  approximates  the  sample  function  if  a very  large  number  of 
systems  are  on  test  simultaneously,  and  the  individual  estimators 
are  averaged ; 

(2)  Secondly,  we  shall  examine  whether  pooling  the  estimates  for 
many  identical  systems  on  test  provides  any  improvement,  by 
looking  at  the  limiting  values  of  these  pooled  statistics. 
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Naturally,  a more  complete  analysis  would  examine  the  distribution 
of  each  statistic  with  a finite  number  of  systems  on  test;  but,  as  we 
shall  see,  the  (undesired)  bias  of  these  estimators  negates  any  more 
detailed  analysis. 
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3.  THE  FAILURE  RATE  ESTIMATOR 

If  the  estimator  (1)  is  used,  its  mean  value  is: 

(5)  = M(t)/t 

where  M(t)  = E{N(t)}  is  the  ordinary  renewal  (counting)  function, 
given  by: 


(6) 


M(t)  = F (t)  + 


M(t  - x)dF(x)  . 


Many  analytic  results  are  known  about  M(t)  , such  as: 


(7) 

(8) 

(9) 


(t/m  ) - 1 1 M(t)  < F(t)/F(t) 

2 t2 

M(t)  « f (0)  • t + [f * (0)  + fZ(0)]  y (t  - 0) 


with  many  more  results  known  for  specific  F , or  special  shape  assumptions, 
such  as  IFR,  IFRA,  NBU,  etc. 

However,  here  we  merely  note  that  (6)  will  only  be  (m^)  ^ for  all 
t in  the  special  case  of  exponential  lifetimes.  In  more  general  cases 
M(t) / t begins  at  a value  f (0)  (the  intercept  of  the  lifetime  density) 
and  can  either  grow  or  decline  C-f'  (0)  > f2(0)]  for  small  values  of  t . 
For  large  values  of  t , M(t)/t  can  either  approach  (m^)  1 from  above 
or  below  depending  upon  whether  the  process  is  more  or  less  "regular" 


than  the  exponential. 
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In  short,  the  transient  behavior  of  the  mean  value  of  N(t)/t  over 
a few  average  lifetimes  can  exhibit  either  apparant  reliability  "growth" 
or  "decay,"  even  though  the  underlying  process  is  stationary . 

If  S systems  were  on  test,  and  each  reported  N failures 
(i  = 1,2,  ...,  i)  , a pooled  estimator  would  be: 


(10) 


U t) 


£ Ni(t)  a . s . M(t) 
S • t t 


as  S ->  00  , giving  the  same  result. 
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4.  THE  MEAN  VALUE  OF  NAIVE  MTBF  ESTIMATOR 

The  average  value  of  the  naive  MTBF  estimator  (2), 

(11)  ei(t)  = E{p1(t)}  = E{[YN(t)/N(t)]  | N(t)  > 0} 

has  interesting  properties.  By  direct  argument 


(12)  Pr  (N(t)  = n ; Y^j  e (y,y  + dy]}  = fn  (y)F(t  - y)dy  , y e [0,t] 
n* 

where  f is  the  n-fold  convolution  of  f , so  that: 

^ fn  (y)F(t  - y)dy 
^ • 

The  numerator  can  be  simplified  using  LaPlace  (-Stieltjes)  transforms, 
giving  the  surprising  results: 

J yf(y)dy  t 

(14)  ex(t)  = F(t)  = J f1  " fOo]  dy  = ! X c t}  , 


the  truncated  mean  of  the  underlying  distribution. 

From  this,  we  observe: 

(1)  The  average  value  of  the  naive  estimator  will  always  exhibit 
apparent  "reliability  growth"  as  t increases; 

(2)  Only  if  X has  a finite  range  [0,R]  will  e^(t)  achieve 
its  ultimate  value,  m^  , for  t >_  R . 


More  specifically,  for  small  t , 


e^t) 


* I + 12  [t$]  '2  * «<°>  * °> 


(15) 


i rf,,(Q>l 

36  L f’(0)J 


(f  CO)  = 0)  (f’(0)  + 0) 


that  is,  the  apparent  growth  in  the  estimator  depends  only  on  the  shape 

of  the  density  f near  the  origin,  not  on  any  specific  values. 

For  large  t , the  behavior  of  e^(t)  depends  upon  the  tails  of  F . 

— 1+cx 

Specifically,  for  algebraic  tails,  F(t)  « A/t  for  large  t , 
e^t)  ss  m^  - A(1  + a)/(ata)  . 

The  transient  behavior  of  (14)  can  be  quite  long.  For  exponential 
lifetime,  F(y)  = e ^ , we  find: 


(16) 


e (t)  = I [1  - (1  + Xt)e~U] 

V ; a r,  -At, 


[1  - e 


From  Figure  2 (solid  line  #1)  we  see  that  there  is  a significant  bias 
in  the  naive  MTBF  for  testing  intervals  less  than,  say,  four  MTBFs. 


5.  THE  CLEVER  MTBF  ESTIMATE 


The  average  of  (3)  cannot,  in  general,  be  put  into  simple  form: 

a;)  -t  I rrrp.(t), 

n=0 

where  Pn(t)  is  the  counting  distribution  associated  with  N(t)  . 
However,  since 


(18) 


P (t)  = F (t)  - F 
n 


(n+1)* 


(t)  , 


n*  n* 

where  F is  the  distribution  corresponding  to  f , we  have  the 

equivalent  result 


(19) 


,(t)  = t - t l 


n 


n=l 


n(n  + 1) 


F (t) 


For  small  t , we  have  the  approximate  result 


(20) 


e2(t)  « t 


f(0)  r2 
2 


(0)  . f (0)  ..3 
4 12  C 


Since  N(t)  ■+  00  and  -►  1/m^  with  probability  one  as  t •>  00  , 

e2(t)  approaches  m^  . We  suspect  this  growth  is  monotonic  (apparent 
"reliability  growth")  in  general,  but  have  not  been  able  to  prove  it. 
The  exponential  case  can  be  summed  explicitly,  giving: 


(21) 


e2(t)  = y [1  - e-Xt]  . 


Figure  2 (Solid  line  //2)  shows  that  this  estimator  grows  more  rapidly 
than  (16),  but  still  shows  significant  bias.  If  we  neglect  the  cases 
where  N(t)  = 0 , 
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(22) 

which 


{ N (t)  + 1 


N(t)  > 0 


A 


-At 

e (1 


+ Xt)l 


1 - e 


-At 


2 eL(t) 


shows  in  what  sense  the  reliability  engineer  was  clever  about 


the  incomplete  sample. 
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6.  THE  SOPHISTICATED  MTBF  ESTIMATE 

The  average  of  the  total-time-on-test  estimator  (4)  is  also  difficult 
to  simplify. 


t l ^Pn(t) 

(23)  e3(t)  = E{p3(t)}  = E {3^-  | N(t)  > 0}  = —~(T) 

Using  (18) , we  obtain 


e3(t)  = 


. ‘h  ' X roVir Fn,(t>] 


For  small  (t)  , we  obtain 


e3(t)  a t - f (0)  -y-  (f (0)  + 0) 


f"(0)  t 


!Bt"V(3)  3 (f'(0)  * 0) 


Again,  e3(t)  •>  m^  as  t -*■  00  , probably  monotonically . 
The  exponential  case  can  be  expressed  as: 


e3(t) 


Ut)2e“Xt 


1 - e 


(At)J' 

jj! 


1 - e 


and  is  shown  in  Figure  2,  (Solid  line  #3). 

Thus  the  mean  of  the  sophisticated  MTBF  estimator  grows  more  rapidly 
than  the  other  two,  but  "over  shoots"  the  true  value  by  more  than  30% 
at  about  four  MTBFs ! There  it  exhibits  very  slow  "reliability  decay," 
being  still  1.056/A  at  20  MTBFs! 


c#  UB3W 


FIGURE  2:  Growth  of  three  estimators  of  X (MTBF ) with  testing  interval  for 

exponential  samples.  Solid  line  shows  mean  value  of  estimator  with  single 
system;  dashed  lines  show  limiting  value  of  pooled  tests  of  many  systems. 
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7 . POOLED  ESTIMATORS 

It  might  seem  that  pooling  the  estimators  for  many  simultaneous 
systems  on  test  might  eliminate  some  of  the  above  difficulties.  We 
examine  the  limiting  case  when  the  number  of  systems  is  large. 

For  the  naive  estimator,  assume  that  only  N^t)  > 0 and  the 
corresponding  epochs  are  reported  from  each  system.  Then 


(27) 


^(t) 


i v« 

i=l  i a.s. 


E{  Y. 


I (N,(t)  > 0) 
i-1  1 


N(t) 


N(t)  > 0} 


E{ N(t)  | N(t)  > 0} 


and  it  can  be  shown  that  in  the  exponential  case: 

(28)  »!<*>  -it1'  ' 

From  Figure  (2)  (Dashed  line  #1) , we  see  that  this  is,  in  fact. 

Worse  than  the  mean  of  the  unpooled  estimator. 

For  the  clever  estimator,  instead  of  averaging  (3)  over  all  systems, 
one  would  pool  both  the  total  interval  and  the  number  of  (complete  and 
incomplete  samples)  including  cases  where  N^(t)  is  zero,  so: 

(29)  i2(t)  = — 

l (N 
i-1 

in  the  exponential  case. 

w2(t)  = I [xt  + l] 


St 


a.s. 


(t)  + 1) 


| l N±(t)  + 1 


M(t)  + 1 


(30) 


which,  in  Figure  2 (Dashed  line  it 2)  is  also  well  below  the  mean  value 
for  a single  system. 

Finally,  for  the  sophisticated  estimator,  who  receives  only  data 
with  at  least  one  failure  from  each  system,  we  can  argue  that,  in  the 
limit : 


(31)  u3(t) 


St a.s.  < t tF (t) 

S E{N(t)  | N(t)  > 0}  " M(t)  ‘ 

l (N.(t)  > 0) 

1=1 


In  the  exponential  case 

(32)  h3(t)  a ~ (1  - e‘Xt) 

which  is  identically  (21). 

Although  pooled  estimates  from  several  systems  on  test  no  doubt 
have  greatly  reduced  variance,  we  must  conclude  that  they,  in  fact, 
exhibit  worse  "reliability  growth"  bias  than  the  corresponding  mean 


estimators. 


8.  CONCLUDING  REMARKS 


In  concluding,  it  is  perhaps  appropriate  to  mention  some  other 
observations  about  the  reliability  growth  literature: 

(1)  There  often  seems  to  be  confusion  between  the  growth  or  decline 
of  hazard  rate  function,  h(t)  = f(t)/F(t)  , and  the  under- 
lying mechanisms  (if  any)  of  reliability  improvement.  The 
first  operates  in  "local"  time  (since  repair  or  replacement); 

the  second  operates  in  "global"  time  (since  systems  testing  began). 
We  have  already  seen  how  non-constant  h(t)  can  lead  to  growth 
or  decline  in  the  average  failure  rate,  M(t)/t  , with  no 
underlying  reliability  growth  mechanism. 

(2)  As  pointed  out  in  [2],  if  underlying  reliability  growth  does 
exist  [as  in  a time-varying  Poisson  failure  process],  then 
successive  intervals  are  not  independent,  and,  of  course,  not 
identically  distributed.  Thus,  any  methodology  that  assumes 
growth,  but  uses  standard  Poisson  - or  renewal  - process 
methodology  must  be  viewed  with  suspicion. 

(3)  Some  literature  speaks  as  if  total  operational  time  is  sig- 
nificant - that  is,  a test  over  5,000  for  1,000  hours  each. 

In  fact,  we  have  seen  that  parallel  testing  may  stabilize  the 
estimator,  but  does  not  remove  the  transient  bias  over  the 
real  time  testing  interval. 
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We  realize  that  more  issues  about  reliability  growth  have  been 
raised  than  resolved.  Removal  of  testing  biases  in  the  stable  case 
is  only  the  first  objective  of  research  in  this  area.  There  is  clearly 
a great  need  for  additional  work  on  extension  of  true,  continuous  reliability 
growth,  as  well  as  investigation  of  the  statistical  confidence  of  these 
estimators  for  different  experimental  protocols.  We  believe  a correct 
formulation  of  this  problem  must  use  a Bayesian  point  of  view,  and  in- 
corporate the  costs  of  experimentation  into  the  design  decision  process. 

Only  in  this  way  will  performance  guarantee  programs  be  placed  on  a sounder 
analytical  basis. 


19 


REFERENCES 


[1]  Annual  Reliability  and  Maintainability  Symposium,  1978  Proceedings, 

Los  Angeles,  California,  January  17-19,  1978,  IEEE,  N.Y.C. 

[2]  Ascher,  H.  and  H.  Feingold,  "Is  There  Repair  After  Failure?," 

in  [1],  pp.  190-197. 

[3]  Balaban,  H.  S.  and  M.  A.  Meth,  "Contractor  Risk  Associated  with 

Reliability  Improvement  Warranty,"  in  [1],  pp.  123-129. 

[4]  Barlow,  R.  E.  and  R.  A.  Campo,  "Total  Time  on  Test  Processes  and 

Applications  to  Failure  Data  Analysis,"  in  RELIABILITY  AND 
FAULT  TREE  ANALYSIS,  Barlow,  Fussell  and  Singpurwalla , (eds.), 
SIAM,  Philadelphia,  (1975). 

[5]  Barlow,  R.  E.  and  E.  M.  Scheuer,  "Reliability  Growth  During  a 

Development  Testing  Program,"  Teknometrics , Vol.  8,  pp.  53-60, 
(February  1966) . 

[6]  Clarke,  J.  M.  and  W.  P.  Cougan,  "RPM  - A Recent  Real  Life  History," 

in  [1],  pp.  279-285. 

1 7 ] Crow,  L.,  "Reliability  Growth  Modelling,"  Technical  Report  No.  55, 

U.  S.  Army  Material  Systems  Analysis  Agency,  Aberdeen 
(August  1972). 

[8]  Hauter,  A.  J.  and  C.  W.  Strempke,  "Tacan  RIW  Program,"  in  [1], 

pp.  62-65. 

[9]  Jewell,  W.  S.,  "Bayesian  Life  Testing  Using  the  Total  Q on  Test," 

in  THE  THEORY  AND  APPLICATION  OF  RELIABILITY,  WITH  EMPHASIS 
ON  BAYESIAN  AND  NONPARAMETER  METHODS,  Vol.  1,  pp.  49-66, 
Academic  Press,  New  York  (1977). 

[10]  Lakner,  A.  A.,  R.  T.  Anderson  and  A.  DiGianf ilippo , "Cost  Effective 

Reliability  Testing,"  in  [1],  pp.  271-278. 

[11]  Lilius,  W.  A.,  "Reliability  Growth  Planning  and  Control,"  in  [1], 

pp.  267-270. 

[12]  Newman,  D.  G.  and  L.  D.  Nesbitt,  "USAF  Experience  with  RIW,"  in  [1], 

pp.  55-61. 

[13]  Shurman,  M.  B. , "Time  Dependent  Rates  for  Jet  Aircraft,"  in  [1], 

pp.  198-203. 

[14]  Weiss,  H.  K. , "Estimation  of  Reliability  Growth  in  a Complex  System 

with  Poisson-Type  Failure,"  Operations  Research,  Vol.  4, 
pp.  503-628,  (October  1956). 


